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Introduction 

The last major review of the achievement outcomes 
of comprehensive school reform (CSR) models was 
conducted in 2003. It concluded the following: 

• The overall evidence base supporting CSR was 
emerging and still quite limited. 

• Many studies were biased because they did not 
include control groups and they were conducted by 
the CSR developers themselves. 

• Several CSR models had accumulated particularly 
strong evidence that they were capable of improving 
achievement across a variety of school contexts. 

Despite the growing evidence base supporting CSR, the 
program was discontinued by the federal government 
in 2007. Now, six years after the 2003 meta-analysis, 
the study's lead author, Geoffrey Borman, Ph.D., revisits 



the results and interprets how the policy and research 
landscape has evolved over the years. 1 

The Study 

Borman, G. D., Hewes, G. M., Overman, L. T., & Brown, S. 
(2003). Comprehensive school reform and achievement: 

A meta-analysis. Review of Educational Research, 73, 
125-230. 

Methodology 

In 2003, Geoffrey D. Borman and colleagues published 
a comprehensive review of all known studies of the 
achievement effects of the 29 most widely implemented 
CSR models. The meta-analysis allowed the researchers 
to summarize quantitatively all information about the 
effectiveness of the use of CSR models as a national 
policy option for raising achievement. This study 
was particularly salient, given the premise of the 



1 Editor’s note: Funding for the Comprehensive School Reform Program began with federal appropriations in 1998. The program was signed into law as 
Title I Part F of the Elementary and Secondary Education Act, as reauthorized by the No Child Left Behind Act of 2001. Federal allocations were made 
to states that, in turn, funded local education agency (LEA) grant applicants. The aim of the program was to “provide financial incentives for schools 
to develop comprehensive school reforms, based on scientifically based research and effective practices that include an emphasis on basic academics 
and parental involvement so that all children can meet challenging State academic content and academic achievement standards” (Title I, Part F, 
Section 1601). LEAs were permitted to adopt an externally developed model or develop their own based on 11 required components. 
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now-discontinued federal CSR program to 
fund schools' implementations of only those 
reform models having rigorous evidence of 
effectiveness. The 2003 meta-analysis provides 
needed insight into the outcomes of this federal 
program. The researchers' preliminary analysis 
accomplished the following: 

• Characterized the overall quality of the 
research evidence. 

• Empirically identified and quantified the 
potential methodological biases in the 
literature. 

• Revealed the common characteristics of CSR 
programs that made a difference in terms of 
student achievement outcomes. 

• Explored differences in achievement effects 
associated with varying contexts (e.g., the 
grade level and the subject area targeted 
by the reform, and the poverty level of the 
school implementing the reform). 

After characterizing the overall CSR research 
base and empirically identifying its potential 
methodological biases, the researchers' second 
objective was to assess the effectiveness 
of each of 29 CSR models. The question of 
effectiveness was addressed by focusing on only 
the subgroup of studies that provided the best 
evidence for evaluating the effects of each of 
the CSR models. 

In Brief 

After completing their review, the researchers 
came to several key conclusions regarding: 

(a) the quantity and quality of the research 
supporting CSR; (b) the overall "policy effect" 
of the federal government's CSR program; 

(c) variations in the achievement effects of CSR 
models across different types of schools and 
contexts; and (d) a determination of which 
of the 29 models, in particular, appeared to 
hold the most promise for improving student 
achievement. Below, the key findings are 
summarized, and lead researcher, Borman, 



discusses how the more recent research and 
policy over the past six years has informed the 
current knowledge base for CSR. 

First, in 2003, CSR was still an emerging field. 
Some models were at an early stage of program 
development that had not yet demanded third- 
party evaluations and more costly and difficult 
control-group studies. Nevertheless, there were 
clear limitations on the overall quantity and 
quality of studies supporting the achievement 
effects of many of the 29 models reviewed. 

More than 40 percent of the analyses of CSR 
effects were nonindependent evaluations that 
had been performed by the developers, and 
only about half of the analyses used some type 
of control group. Only seven studies of three 
CSR models (about 3 percent of all studies of the 
achievement effects associated with CSR) had 
generated evidence from that which researchers 
term the "gold standard" of causal inference, the 
randomized experiment. These limitations were 
important, because Borman and his colleagues 
concluded that evaluations of the models that 
were performed by the developers themselves 
and that did not use control groups were prone 
to potential biases that seemed to inflate the 
estimates of the models' effects. 

Second, Borman and his colleagues found 
that the overall effects of CSR models were 
statistically significant and meaningful, and they 
appeared to be greater than the effects of other 
interventions — such as Title I targeted assistance 
and schoolwide programs — that have been 
designed to serve similar purposes and student 
and school populations. Overall, students from 
CSR schools could be expected to score one 
eighth of a standard deviation, or 2.5 normal 
curve equivalent (NCEs), higher on achievement 
tests than control students in non-CSR schools. 
These overall effects, though, were highly 
variable and should be viewed as averages 
found across a wide array of reform models and 
schools that were evaluated in a variety of ways. 
The overall effect size is a good indicator of the 
expected effects of CSR across a large number 
of schools. Policymakers may expect to find CSR 
effects of about 2.5 NCE points in similar studies 
of national or large districtwide samples of CSR 




schools. But the effects for individual schools 
and the effects for individual reform models are 
likely to vary. 

Third, the authors found that substantial 
differences in the effectiveness of CSR models 
were largely a result of program-specific and 
school-specific differences in implementation. 

One important factor was the number of years 
that the reform had been implemented. In 
those schools implementing CSR models for 
five years, achievement effects were twice as 
large as the overall effect of 2.5 NCEs; these 
effects were even greater for schools with longer 
implementation histories. In addition, the choice 
of the CSR model itself was the source of a great 
deal of variation in the effectiveness of CSR. 
Interestingly, though, the 1 1 federally mandated 
components of CSR and the poverty level of 
the school had little to do with the effectiveness 
of the programs. That is, regardless of which of 
the 1 1 federally defined components of a CSR 
model that a school chose to emphasize in its 
implementation and regardless of the poverty 
context of the school, the achievement outcomes 
remained very similar. The fidelity and duration of 
implementation of any model was more related 
to outcomes than the particular model, model 
component, or school poverty level. 

Finally, the three programs meeting the highest 
standard of evidence — Direct Instruction, the 
School Development Program, and Success for 
All — were the only CSR models to have clearly 
established, across varying contexts and varying 
study designs, that their effects were relatively 
robust and that the models, in general, could be 
expected to improve students' test scores. The 
outcomes varied considerably by reform model. 
In most cases, however, the research base for 
each CSR model was too small to generate 
reliable estimates of the model's expected 
effects. In some cases, promising and highly 
promising models were beginning to emerge. 
Expeditionary Learning Schools Outward 
Bound, Modern Red Schoolhouse, and Roots 
& Wings were all on the brink of establishing 
strong research bases. The models meeting 
the standard for the "Strongest Evidence of 
Effectiveness" category were distinguished from 



these models and others by the quantity and 
generalizability of their outcomes; the quality 
of this evidence (for instance, six of the seven 
randomized experiments and many high-quality 
quasi-experimental control-group studies have 
been conducted on the models achieving 
the highest standard of evidence); and the 
statistically reliable effects on achievement. 

CSR Research and 
Policy Since 2003 

Since 2003, education research and policy 
stakeholders have remained committed to 
the idea of promoting the use of rigorous 
methods — most notably randomized trials — for 
establishing the effectiveness of educational 
programs and practices. Through a variety 
of federal initiatives, including No Child Left 
Behind (NCLB), the former Reading First 
program, and the establishment of the Institute 
of Education Sciences (the research arm of the 
U.S. Department of Education), evidence-based 
education policy has flourished. However, the 
CSR program — once one of the most prominent 
efforts to link research to practice — has not been 
supported by the federal government, politically 
or fiscally, since 2007. 

Overall, the former federal CSR program had 
somewhat modest effects on achievement. 
However, this situation has frequently been the 
case with large federal education programs, 
including Title I, Reading First, Supplemental 
Educational Services, and others. There is now 
compelling evidence, though, that particular 
CSR models are effective for improving student 
achievement. A growing number of sources for 
systematic reviews of policies and programs 
affecting children exist, including the Campbell 
Collaboration, the What Works Clearinghouse, 
the Best Evidence Encyclopedia, and the 
Coalition for Evidence-Based Policy. 

The 2003 meta-analysis of CSR programs and 
these more recent compendia of research 
evidence have identified a number of models 
with rigorous evidence of success, such as 
Accelerated Schools, Career Academies, Direct 
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Instruction, School Development Program, Success 
for All, and Talent Development High Schools. 2 
With the continued federal focus on research-based 
solutions for improving America's schools and the 
expanding number of CSR models with evidence of 
effectiveness, why has CSR so abruptly fallen out 
of favor? 

On the face of it, the discontinuation of the federal 
CSR grant program — which allocated as much as 
$300 million to support school reform during 2002 — 
was a major blow to the CSR movement. However, 
this loss of funds did not necessarily represent the 
death knell for CSR. Schools with concentrations of 
poor children are able to garner sufficient resources 
to implement CSR models by reallocating existing 
supplemental funds and personnel from federal 
Title I programs, state poverty supplements, special 
education, and other sources. In this way, many 
high-poverty schools can afford even the most costly 
CSR models by trading in their largely remedial 
approaches of the past, most often represented by 
federal and state Title I programs, for new designs 
that will enable them to implement research-based 
schoolwide reform programs. This method of 
resource reallocation can make implementations of 
CSR programs essentially "costless." 



2 These compendia of research evidence on various school-based programs and practices can be found on the Internet at the 
following locations: http://www.campbellcollaboration.org/ (Campbell Collaboration); http://ies.ed.gov/ncee/wwc/ (What Works 
Clearinghouse); http://www.bestevidence.org/ (Best Evidence Encyclopedia); and http://www.evidencebasedprograms.org/ 
static/index.htm (Coalition for Evidence-Based Policy). 
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Bottom Line 

In terms of increased student achievement, CSR 
appears to: (a) have an overall positive effect; 

(b) be effective whether a school is relatively lower 
or higher on poverty measures; (c) increase its 
effectiveness for an individual school the longer it is 
implemented there; (d) include a variety of models, 
with a number of them generating strong evidence 
of effectiveness over the years; and (e) depend for 
its effectiveness more on program implementation 
than on whether it contains a predetermined set of 
federally required components. Schools and district 
continue to employ a number of CSR models and 
fund them with Title I and other monies. 

CSR models can help turn around school 
performance in even the most challenging contexts. 
For this policy option to have its strongest impacts, 
school leaders must carefully select a CRS model 
with strong evidence of effectiveness, be resourceful 
in finding sufficient fiscal support to sustain the 
reform, work hard to implement the design as 
intended, and stick with it for at least five years. 




